CombTransformers: Statement-Wise Transformers for Statement-Wise Representations

This study presents a novel category of Transformer architectures known as comb transformers, which effectively reduce the space complexity self-attention layer from quadratic to sub-quadratic level. is achieved by processing sequence segments independently and incorporating X -word embeddings merge cross-segment information. The reduction in attention memory requirements enables deployment deeper architectures, potentially leading more competitive outcomes. Furthermore, we design an abstract syntaxtree (AST)-based code representation exploit transformer properties. To explore potential our approach, develop nine specific instances based on three popular architectural concepts: funnel, hourglass, encoder-decoder. These are subsequently trained code-related tasks: method name generation, search, summarization. tasks encompass range capabilities: short/long generation classification. In addition proposed also evaluate several baseline for comparative analysis. Our findings demonstrate that transformers match performance baselines frequently perform better.